{% extends "base.html" %}
{% block css %}<link rel="stylesheet" type="text/css" href="{{settings.URL_ROOT}}media/css/codewiki.css" />{% endblock %}

{% block title %}Scope index{% endblock %}

{% block content %}

<h1>Metroscope/codewiki project</h1>

<h4><em><a href="{{settings.URL_ROOT}}scope">Back to index</a></em></h4>

<p>There are two independently functioning parts to this project.  
At the moment they are built into this same prototype system, but 
they should be running on entirely different servers.</p>

<p>The first part is the set of scrapers and parsers of the data presented on council 
webpages and from other official sources whose data would be useful 
for many different downstream applications if only it was in a suitably structured form.</p>

<p>The Codewiki is a system for managing these scrapers and parsers 
where the work can be shared among community of programmers so 
that there will never again be a situation where two organizations 
need to write and maintain their own parsers for the same data.  
It also has the side-effect of making visible this essential 
aspect of these projects to those who do not program, but  
may be responsible for creating the problem in the first place with their poorly 
structured webpages and lack of standard database APIs.</p>

<p>The second part is the database into which the scrapers and parsers 
maintained on the Codewiki submit their data.   
One particular application of this data is the Metroscope,  
very similar to the <a href="http://blog.brightkite.com/2008/11/19/introducing-the-brightkite-wall/">Brightkite wall</a>, 
which is getting some publicity at the moment.  
There will hopefully be countless other expressions of the database in different 
and more imaginative applications, but it is essential, however, to 
promote one specimen application in the first place to motivate the 
development of all these scrapers and parsers which we would like 
to come into being.</p>

<h2>Codewiki tools</h2>

<p>This is the part which needs to be funded and managed through development.
The features that need to be created are easy to list:</p>

<ul>
<li>A secure server infrastructure that can safely run user-generated scripts</li>
<li><a href="http://marijn.haverbeke.nl/codemirror/">Syntax highlighting</a> in the codewiki editor</li>
<li>Saving and running without refreshing the screen and with streamed script output (AJAX-like) that you can interrupt with Control-C</li>
<li>Version control system for the parsers and edited scraped data</li>
<li>Place name and postcode lookup modules</li>
<li>Support for scripting languages other than Python</li>
<li>Dashboard utilities for scheduling the rerunning and monitoring the output from completed parsers</li>
<li>Scratch databases, instant messenging between logged-in users, forums, etc</li>
</ul>


<h2>Scope tool</h2>

<p>The Metroscope, which uses this data, filters by <b>user</b> and <b>source</b> keys 
so that no filtering need happen to the incoming data.  It will have close parallels with the 
<a href="http://wordaligned.org/articles/turing-tests-and-train-trackers">voice announcement system</a>
in use in busy train stations where the events (train departures) have to be made several times and 
in good order with enough details.</p>

<p>The <b>metroscope</b> with its constantly feeding set of events would be time and location specific, 
with some input from the news agenda or trend-spotting, in particular for notifying static events, such as 
the location of a museum.  Also, the clicking habits of your neighbours on the links that appear in their 
metroscopes would alter the amount you would see of those events.  A seamless integration of 
friends' interested in features of <a href="http://en.wikipedia.org/wiki/Upcoming">Upcoming</a> 
would be desirable.  Import of events from that portal allows for user-generated events to be 
added, not only the government-sanctioned ones.</p>

<p>Information is targetted by what you can get to.  That's what makes these events and news instances special: 
you can physically interact with them.  It's not about what's interesting, it's about using the internet medium to 
cause real things to happen.  Ideally the metroscope needs to know your exact location and your journey intentions 
(eg the places you pass on your journey home from the office), as well as your modes of transport.  
Someone with a bicycle would have a wider range 
than someone on foot.  Success of the system should be measurable.</p>

<p>Other than that, the data can be output in any form to another database for display on a mashup or on 
some boring static web-page where you have to create the dynamism yourself with eye saccades.</p>

<p>julian@goatchurch.org.uk</p>

{% endblock %}

